Air-side economizer facilitating liquid-based cooling of an electronics rack

ABSTRACT

A cooling apparatus and method are provided for cooling an electronic subsystem of an electronics rack. The cooling apparatus includes a local cooling station, which has a liquid-to-air heat exchanger and ducting for directing a cooling airflow across the heat exchanger. A cooling subsystem is associated with the electronic subsystem of the rack, and includes either a housing facilitating immersion cooling of electronic components of the electronic subsystem, or one or more liquid-cooled structures providing conductive cooling to the electronic components of the electronic subsystem. A coolant loop couples the cooling subsystem to the liquid-to-air heat exchanger of the local cooling station. In operation, heat is transferred via circulating coolant from the electronic subsystem and rejected in the liquid-to-air heat exchanger of the local cooling station to the cooling airflow passing across the liquid-to-air heat exchanger. In one embodiment, the cooling airflow is outdoor air.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. Ser. No. 13/187,561, entitled“Air-Side Economizer Facilitating Liquid-Based Cooling of an ElectronicsRack”, filed Jul. 21, 2011, and which is hereby incorporated herein byreference in its entirety.

BACKGROUND

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both the module and system level. Increased airflow rates are neededto effectively cool high power modules and to limit the temperature ofthe air that is exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power supplies, etc.)are packaged in removable node configurations stacked within anelectronics (or IT) rack or frame. In other cases, the electronics maybe in fixed locations within the rack or frame. Typically, thecomponents are cooled by air moving in parallel airflow paths, usuallyfront-to-back, impelled by one or more air moving devices (e.g., fans orblowers). In some cases it may be possible to handle increased powerdissipation within a single node by providing greater airflow, throughthe use of a more powerful air moving device or by increasing therotational speed (i.e., RPMs) of an existing air moving device. However,this approach is becoming problematic at the rack level in the contextof a computer installation (i.e., data center).

The sensible heat load carried by the air exiting the rack is stressingthe ability of the room air-conditioning to effectively handle the load.This is especially true for large installations with “server farms” orlarge banks of computer racks close together. In such installations,liquid cooling (e.g., water cooling) is an attractive technology tomanage the higher heat fluxes. The liquid absorbs the heat dissipated bythe components/modules in an efficient manner. Typically, the heat isultimately transferred from the liquid to an outside environment,whether air or other liquid coolant.

BRIEF SUMMARY

In one aspect, a method of cooling at least one electronic subsystem ofan electronics rack is provided. The method includes: obtaining acooling apparatus comprising: a local cooling station, the local coolingstation including a liquid-to-air heat exchanger, and ducting fordirecting cooling airflow across the liquid-to-air heat exchanger; atleast one cooling subsystem for association with the at least oneelectronic subsystem, one cooling subsystem of the at least one coolingsubsystem to provide cooling to a respective electronic subsystem of theat least one electronic subsystem, the one cooling system comprising atleast one of a housing facilitating immersion cooling of one or moreelectronic components of the respective electronic subsystems, or aliquid-cooled structure providing conductive cooling of one or moreelectronic components of the respective electronic subsystem; and atleast one coolant loop for coupling the one cooling subsystem to theliquid-to-air heat exchanger of a respective local cooling substation;disposing the electronics rack and the respective local cooling stationadjacent to each other, and employing the one coolant loop to couple influid communication the one cooling subsystem associated with therespective electronic subsystem and the liquid-to-air heat exchanger ofthe respective local cooling station; and establishing cooling airflowthrough the ducting and across the liquid-to-air heat exchanger, andcirculation of coolant through the liquid-to-air heat exchanger, thecoolant loop and the one cooling subsystem, wherein heat is transferredvia the circulating coolant from the respective electronic subsystem andrejected in the liquid-to-air heat exchanger of the respective localcooling station to the cooling airflow passing across the liquid-to-airheat exchanger.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1. depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 is a schematic of one embodiment of a data center comprising oneor more electronics racks and a cooling apparatus, in accordance withone or more aspects of the present invention;

FIG. 3 is an enlarged, partial schematic of the cooling apparatus andelectronics rack depicted in FIG. 2, in accordance with one or moreaspects of the present invention;

FIG. 4 is a partial top plan view of the data center of FIG. 2, inaccordance with one or more aspects of the present invention;

FIG. 5A is a schematic of an alternate embodiment of a data centercomprising a cooling apparatus, including an air-side economizerfacilitating liquid-based cooling of one or more associated electronicsracks, in accordance with one or more aspects of the present invention;

FIG. 5B is an enlarged, partial schematic of the cooling apparatus andassociated electronics rack of FIG. 5A, in accordance with one or moreaspects of the present invention;

FIG. 6A is a schematic of an alternate embodiment of a data centercomprising a cooling apparatus, including an air-side economizerfacilitating liquid-based cooling of one or more electronics racks, inaccordance with one or more aspects of the present invention;

FIG. 6B is an enlarged, partial schematic of the cooling apparatus andassociated electronics rack of FIG. 6A, in accordance with one or moreaspects of the present invention;

FIG. 6C is another enlarged, partial schematic of the cooling apparatusand associated electronics rack of FIG. 6A, in accordance with one ormore aspects of the present invention;

FIG. 7 depicts one embodiment of processing for controlling operation ofa cooling apparatus, such as depicted in FIGS. 2-6C, in accordance withone or more aspects of the present invention;

FIG. 8A depicts one embodiment of processing for controlling operationof the evaporative cooling system of a cooling apparatus, such asdepicted in FIGS. 2-6C, in accordance with one or more aspects of thepresent invention;

FIG. 8B is a schematic of one embodiment of a controllable evaporativecooling system employed in a cooling apparatus, such as depicted inFIGS. 2-6C, in accordance with one or more aspects of the presentinvention; and

FIG. 9 depicts one embodiment of a process for controlling acontrollable recirculation fan coupling the airflow exhaust plenum tothe cooling airflow supply plenum of a cooling apparatus, such asdepicted in FIGS. 2-6C, in accordance with one or more aspects of thepresent invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronic system, and may be, for example, astand-alone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise aportion of an electronic system, a single electronic system, or multipleelectronic systems, for example, in one or more sub-housings, blades,books, drawers, nodes, compartments, etc., having one or moreheat-generating electronic components disposed therein. An electronicsystem(s) within an electronics rack may be movable or fixed relative tothe electronics rack, with the rack-mounted electronic drawers of amulti-drawer rack unit and blades of a blade center system being twoexamples of systems (or subsystems) of an electronics rack to be cooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, a computer system or other electronic systemrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesand memory support dies. As a further example, the electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier.

Unless otherwise specified herein, the terms “liquid-cooled structure”and “liquid-cooled cold plate” refer to thermally conductive structureshaving one or more channels (or passageways) or chambers formed thereinor passing therethrough, which facilitate flow of coolant therethrough.In one example, tubing may be provided extending into or through theliquid-cooled structure (or liquid-cooled cold plate).

As used herein, “liquid-to-air heat exchanger” means any heat exchangemechanism characterized as described herein through which liquid coolantcan circulate; and includes, one or more discrete liquid-to-air heatexchangers coupled either in series or in parallel. A liquid-to-air heatexchanger may comprise, for example, one or more coolant flow paths,formed of thermally conductive tubings (such as copper or other tubing)in thermal or mechanical contact with a plurality of air-cooled coolingfins. Size, configuration and construction of the air-to-liquid heatexchanger can vary without departing from the scope of the inventiondisclosed herein. Further, as used herein “data center” refers to acomputer installation containing, for example, one or more electronicsracks to be cooled. As a specific example, a data center may include oneor more electronic racks, such as server racks.

One example of the coolant employed herein is water. However, theconcepts disclosed herein are readily adapted to use with other types ofcoolant. For example, one or more of the coolants may comprise a brine,a dielectric liquid, a fluorocarbon liquid, a liquid metal, or othersimilar coolant, or refrigerant, while still maintaining the advantagesand unique features of the present invention.

Reference is made below to the drawings, which are not drawn to scale tofacilitate understanding thereof, wherein the same reference numbersused throughout different figures designate the same or similarcomponents.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, chilled air enters the computer room viaperforated floor tiles 160 from a supply air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the back (i.e., air outlet sides130) of the electronics racks. Each electronics rack 110 may have one ormore air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic devices within thesubsystem(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the computerinstallation. The conditioned and cooled air is supplied to plenum 145by one or more computer room air-conditioning (CRAC) units 150, alsodisposed within the data center 100. Room air is taken into each airconditioning unit 150 near an upper portion thereof. This room air maycomprise in part exhausted air from the “hot” aisles of the computerinstallation defined, for example, by opposing air outlet sides 130 ofthe electronics racks 110.

Due to the ever-increasing airflow requirements through electronicsracks, and the limits of air distribution within the typical data centerinstallation, novel cooling apparatuses and methods are needed.Disclosed herein, therefore, are cooling apparatuses and methodscombining a liquid-based cooling approach with, for example, an air-sideeconomizer, for extracting heat from the liquid-based cooling approach.FIGS. 2-9 illustrate various embodiments of a data center implementingsuch cooling apparatuses and methods for cooling electronic subsystemsof one or more electronics racks, in accordance with one or more aspectsof the present invention.

As noted initially, data center equipment may house several hundred, oreven several thousand heat-generating electronic components, such asmicroprocessors. Cooling computer and telecommunications equipment roomscan be a major challenge. In fact, cooling has been found to contributeabout one-third of the energy use of a typical IT data center.

In a conventional data center, sub-ambient temperature, refrigeratedwater leaves a chiller plant evaporator and is circulated through one ormore CRAC units (see FIG. 1) using building chilled water pumps. Thiswater carries heat away from the air-conditioned, raised floor room thathouses the IT equipment, and rejects the heat into the refrigerationchiller evaporator via a heat exchanger. The refrigeration chilleroperates on a vapor-compression cycle that consumes compression work(compressor). The refrigerant loop rejects the heat into a condenserwater loop using another chiller heat exchanger (condenser). A condenserpump circulates water between the chiller condenser and the air-cooled,evaporative cooling tower. The air-cooled cooling tower uses forced airmovement and water evaporation to extract heat from the condenser waterloop, and transfer it into the ambient environment. Thus, in such a“standard” facility cooling design, the primary cooling energyconsumption components are: the server fans; the computer roomair-conditioning (CRAC) unit blowers; the building chilled water (BCW)pumps; the refrigeration chiller compressors; the condenser water pumps;and the cooling tower fans.

As a departure from this typical cooling approach, a cooling apparatusand method are disclosed herein which provide energy efficient coolingof electronic subsystems, such as servers, and other informationtechnology equipment, of a data center. As described below, in oneembodiment, outdoor air is drawn in and conditioned as a cooling airflowto which heat is rejected from one or more liquid-cooled electronicsubsystems of one or more electronics racks within the data center. Theoutdoor air may be used “as is”, or may be conditioned using, forexample, a filter and an evaporative cooling system in which water issprayed onto a porous media, while the outdoor air is forced through themedia, thus evaporating the water resident on the surfaces of the porousmedia directly into the air. Such evaporative cooling system, whichreduces the dry bulb temperature of the air, may comprise a commerciallyavailable system, such as the evaporative cooling systems available fromMunters Corporation, of Amesbury, Mass., U.S.A. Using such anevaporative cooling system can reduce the temperature of the outdoor airdrawn into the cooling apparatus to be close to the air's wet bulbtemperature. Thus, in hot summer months, the use of evaporative cooling(for example, at the inlet of a cooling airflow supply plenum of thecooling apparatus) can provide significant reduction in the intake airtemperature, that is, significant reduction of the temperature of theoutdoor air used for indoor cooling, as described hereinbelow.

Unfortunately, it can be problematic to use outdoor air directly insidea data center room, even with further cooling using evaporative methods.Outdoor air can often possess several undesirable attributes, such ascontaining particulate pollution or chemical or gaseous pollution, whichboth can be extremely harmful to electronic hardware. Thus, disclosedherein is a data center cooling system that possess the beneficialenergy-saving attributes of an air-side, economizer-based coolingapproach, but which also provides protection from the harmful propertiesof the outdoor air.

FIGS. 2 & 3 depict a data center, generally denoted 200, comprising oneembodiment of such a cooling apparatus. As shown, cooling apparatus 200comprises an air-side economizer 201 and liquid-based cooling of one ormore electronic subsystems 220 of one or more electronics racks 210. Inthe depicted embodiment, data center 200 includes multiple local coolingstations 240, each of which is (in one embodiment) associated with, butfree-standing from, a respective electronics rack 210 comprising one ormore electronic subsystems to be cooled. Local cooling station 240includes (in one embodiment) a vertically-extending, liquid-to-air heatexchanger 243 and supply and return ducting 241, 242 for directing acooling airflow 244 across liquid-to-air heat exchanger 243.

The liquid-based cooling aspect of the cooling apparatus includes, inone embodiment, multiple cooling subsystems 219 associated with themultiple electronic subsystems 220, and together forming multipleliquid-cooled electronic subsystems. As shown in FIG. 3, each coolingsubsystem 219 comprises (in this embodiment) a housing 221 whichencloses a respective electronic subsystem 220 comprising a plurality ofelectronic components 223. In this implementation, the electroniccomponents are (by way of example) immersion-cooled in a coolant 224,such as a dielectric coolant. The cooling system is designed for thedielectric coolant to boil in typical operation, generating dielectriccoolant vapor 225. As illustrated, electronic subsystems 220 are angledby providing upward-sloped support rails 222 within electronics rack 210to accommodate the electronic subsystems 220 at an angle. Angling of theelectronic subsystems as illustrated facilitates buoyancy-drivencirculation of coolant vapor 225 between the cooling subsystem 219 andthe liquid-to-air heat exchanger 243 of the associated local coolingstation 240.

In the embodiment depicted in FIGS. 2 & 3, the cooling apparatus furtherincludes multiple coolant loops 226 coupling in fluid communication theliquid-cooled electronic subsystems and a respective portion ofliquid-to-air heat exchanger 243. In particular, multiple sloped tubingsections 300 are provided, as illustrated in FIGS. 2 & 3, passingthrough liquid-to-air heat exchanger 243. Liquid-to-air heat exchanger243 further includes, in this example, a plurality of air-cooling fins310 which may be, in one example, oriented vertically within theliquid-to-air heat exchanger 243. The configuration of the plurality ofair-cooled fins 310 and the multiple sloped tubing sections 300 may bechosen to facilitate the passage of cooling airflow 244 across theliquid-to-air heat exchanger, which in this two-phase example, functionsas a condenser for the coolant vapor circulating therethrough.

In the example of FIGS. 2 & 3, the liquid-cooled electronic subsystemsremain accessible through a front 212 of the electronics rack 210, andmultiple quick connect couplings 246 are provided in association withthe multiple coolant loops 226 to facilitate connection or disconnectionof the respective liquid-cooled electronic subsystem(s) from the localcooling station 240. The multiple coolant loops may include flexibletubing, and quick connect couplings 246 may be any one of various typesof commercially available couplings, such as those available from ColderProducts Co., of St. Paul, Minn., USA, or Parker Hannifin, of Cleveland,Ohio, USA.

As noted, dielectric coolant vapor 225 is buoyancy-driven from housing221 to the corresponding sloped tubing section 300 of liquid-to-air heatexchanger 243, where the vapor condenses and is then returned as liquidto the associated liquid-cooled electronics subsystem. In oneembodiment, the local cooling station 240 is free-standing and separatefrom electronics rack 210, with the liquid coolant loops 226 beingcompleted by positioning electronics rack 210 adjacent to the respectivelocal cooling station 240, and attaching the quick connect couplings.

As illustrated in FIG. 2, an airflow damper 245 is provided to controlthe amount of cooling airflow 244 flowing through supply ducting 241 toliquid-to-air heat exchanger 240. When associated with no electronicsrack or an empty electronics rack 211 that is awaiting the electronicsubsystems (e.g., server units), the respective airflow damper 245 maybe moved to a closed position, as illustrated on the right side of FIG.2, to prevent cooling airflow from passing through the liquid-to-airheat exchanger 243 of the associated local cooling station 240. In theembodiment of FIG. 2, air-side economizer 201 further includes a coolingairflow supply plenum 231 and an airflow exhaust plenum 232. Coolingairflow supply plenum 231 receives outdoor air 230 after being drawnacross a filter 233 via an outdoor air intake fan 234. In the embodimentdepicted, an evaporative cooling system 235 and associated controller236 are provided to selectively cool the outdoor air, depending upon itstemperature, as explained further below.

In one embodiment, cooling airflow 244 is provided in parallel to thesupply ducting 241 of multiple local cooling stations 240 of data center200, and the heated airflow is exhausted via return ducting 242 inparallel from the multiple cooling stations to the airflow exhaustplenum 232. In this embodiment, the cooling airflow supply plenum andairflow exhaust plenum comprise overhead plenums within the data center.

FIG. 4 illustrates one embodiment of a plurality of local coolingstations 240 and an associated row of electronics racks 210, as well asan associated row of empty electronics racks 211. In this embodiment,the liquid-to-air heat exchangers 243 are either door-mounted orpivotally-mounted, liquid-to-air heat exchangers, which facilitatesaccess to, for example, the quick connect couplings (see FIGS. 2 & 3)disposed within or adjacent to return ducting 242. As noted, coolingairflow 244 flows down the respective supply ducting 241, is directedacross the liquid-to-air heat exchanger 243 before being exhausted viareturn ducting 242 to, for example, the airflow exhaust plenum 232 (seeFIG. 2). In the example of FIG. 4, the airflow dampers 245 on the rightside are shown in closed position for the local cooling stations 240associated with the empty electronics racks 211.

FIG. 2 also illustrates a controllable recirculation fan 250, whichcomprises a fan that is selectively controlled (as explained furtherbelow), for example, during winter months, in order to recirculate aportion of the heated airflow exhaust in the airflow exhaust plenum 232directly into the cooling airflow supply plenum 231 for mixing with thecold outdoor air 230, drawn into the cooling apparatus. In winteroperation, the evaporative cooling system 235 would be shut OFF bycontroller 236.

To summarize, in operation, outdoor air 230 is drawn in through, forexample, particulate filter 233, and may be forced through anevaporative cooling system 235, after which it is distributed via thecooling airflow supply plenum 231 to various parts of data center 200.The cooling airflow supply plenum 231 feeds several vertical supplyducts 241 with cooling airflow 244, and this cooling airflow passesthrough the respective liquid-to-air heat exchangers 243, and returnsvia vertical return ducting 242, to airflow exhaust plenum 232, where itis exhausted through an exhaust vent 238 by an exhaust fan 237 to theoutside of the data center. While the intake and exhaust openings to thecooling airflow supply plenum and airflow exhaust plenum, respectively,are shown in FIG. 2 adjacent to each other, in reality, the intake andexhaust openings may be disposed remote from each other. By remotelydisposing the intake and exhaust openings, any mixing of the warmexhaust air with the cooler intake air can be avoided. As explainedfurther below, in winter months, when the outdoor air temperature may bequite cold, the outdoor air temperature may be heated by recirculatingthe warmer exhaust air (as shown in FIG. 2) wherein the controllablerecirculation fan 250 is provided, along with an appropriate opening, tofacilitate controlled heating of the intake air using the warmer exhaustair stream.

As described above, in the embodiment depicted in FIGS. 2-4, theelectronic subsystems (e.g., server nodes) are immersion-cooled, and aredocked using sloped rack rails 222 which angle upwards from front side212 of the rack. The immersion-cooled electronic subsystems containelectronic components 223, such as a printed circuit board,microprocessor modules, and memory devices, that are packaged withinhousing (or container) 221 filled, in this example, with dielectriccoolant 224. The coolant boils where in contact with the electroniccomponents, and the vapor exits the liquid-cooled electronic subsystemvia exhaust tubing of a coolant loop 226 to a respective tubing section300 in the liquid-to-air heat exchanger 243. In this example, theliquid-to-air heat exchanger contains several parallel tubing sections300 through which vapor from different subsystems is condensed, andsubsequently returned back to the respective liquid-cooled electronicsubsystem to repeat the cooling cycle. The sloped nature of theelectronic subsystems facilitates the upwards and natural travel of thevapor to the heat exchanger tube sections, and then the natural downwardreturn of the liquid condensate back to the electronic subsystems. Thus,dielectric coolant circulation (in one example) is via buoyancy-drivenflow.

As noted, an airflow damper may be placed in open position to allowunimpeded flow of cooling airflow through the vertical supply duct andvertical return duct, or may be placed in closed position when an emptyelectronics rack (or no electronics rack) is disposed adjacent to therespective local cooling station. In the closed position, the verticalsupply and return ducts are blocked by the damper, which cuts offairflow through the local cooling station, thus preventing wasting ofpumped airflow within the data center.

Quick connect couplings provided at (for example) inlet and outlet portsof each of the parallel-coupled coolant loops facilitate connection anddisconnection of the respective liquid-cooled electronic subsystems tothe respective local cooling station. In operation, the cooling airflow,which cools the liquid-to-air heat exchanger fins, and thus the insideof the sloped tubing sections through which the dielectric vapor flows,should be at a temperature that is well below the condensationtemperature of the vapor (i.e., the boiling point of the dielectric).This would allow for a temperature difference between the surfacecontacting the vapor and the cooling airflow. In one embodiment, at anytime during operation of the cooling apparatus and electronics rack,there should be a prevailing liquid coolant level within theliquid-immersed electronic subsystem that submerges most of theheat-generating electronic components of the electronic subsystem.

As illustrated in FIG. 4, the shaped profile of the vertical ductsections 241, 242 facilitates the opening of one or more doors of thelocal cooling stations (which are substantially coplanar with the ducts)so as to allow servicing of the electronics rack, from the side of theelectronics rack disposed adjacent to (or in contact with) the localcooling station. In one embodiment, the doors are coplanar and part ofthe duct structure, and can be opened as illustrated in FIG. 4. Once thedoor is opened, the respective liquid-to-air heat exchanger can then berotated about a separate vertical hinge to allow for access to (forexample) the back side of the associated electronics rack, eitherdirectly through an opening in return ducting 242, or by selectiveremoval of return ducting 242.

Note that advantageously, the only cooling energy consumed in thecooling apparatus of FIGS. 2-4 is at the intake and exhaust fans, theselectively operated recirculation fan, and the selectively operatedevaporative cooling system (e.g., the water pump for distributing waterto the evaporative cooling media of the system).

FIGS. 5A & 5B depict an alternate embodiment of a cooling apparatus suchas described above in connection with FIGS. 2-4. In this alternateembodiment, the liquid-cooling approach is modified to incorporate aliquid-cooled structure 530 within (in one embodiment) each of therespective electronic subsystems 220 of the electronics rack 210′. Asillustrated, the liquid-cooled structures 530 are configured, in oneembodiment, to overlie and to provide conduction cooling to one or moreelectronic components 223 of electronic subsystem 220. In oneembodiment, the liquid-cooled structures 530 are water-cooled, withwater being pumped through the respective liquid-cooled structures viaone or more respective node-level pumps 531 disposed, in this example,within or at the electronic subsystem 220.

As illustrated in FIGS. 5A & 5B, the local cooling station 500 includesducting 241, 242, as described above in connection with FIGS. 2-4, whichreceive cooling airflow 244 from a cooling airflow supply plenum andexhaust heated airflow to an airflow exhaust plenum, as described. Thecooling airflow 244 passes through a liquid-to-air heat exchanger 510,which is configured with a plurality of tube sections 512, each of whichis approximately aligned to a respective electronics subsystem 220 ofthe associated electronics rack 210′ disposed adjacent to the localcooling station 500. Liquid-to-air heat exchanger 510 further includes aplurality of air-cooled fins 511 coupled in thermal communication withthe coolant-carrying tube sections 512 of the heat exchanger. Coolantloops 532 are provided to couple the cooling subsystems, in this case,comprising liquid-cooled structures 530 to the respectivecoolant-carrying tube sections 512 of liquid-to-air heat exchanger 510.Quick connect couplings 513, such as the quick connect couplingsdescribed above with reference to FIGS. 2-4, may be employed inassociation with the coolant loops 532 to provide quick connection ordisconnection of a respective liquid-cooled electronic subsystem to thelocal cooling station 500.

In operation, coolant, such as water, or other single-phase liquidcoolant, may be circulated via the node-level pumps through theliquid-cooled structures 530, coolant loops 532, and respectivecoolant-carrying tube sections 512 of the liquid-to-air heat exchanger510. The liquid-cooled structure 530 within a particular electronicsubsystem may comprise a single liquid-cooled structure, or multipleliquid-cooled structures, such as a plurality of liquid-cooled coldplates, or other such conduction-based structures, coupled in fluidcommunication, either in series or in parallel within the liquid-cooledelectronic subsystem. Since there is no buoyancy-driven flow in thisembodiment, the electronic subsystems do not need to be sloped, as inthe embodiment of FIGS. 2-4.

FIGS. 6A-6C depict a further embodiment of a cooling apparatus, inaccordance with one or more aspects of the present invention. Thiscooling apparatus includes an air-side economizer similar to thatdescribed above in connection with FIGS. 2-4, and liquid-cooledelectronic subsystems, such as described above in connection with theembodiment of FIGS. 5A & 5B. In this embodiment, however, the node-levelpumps are removed from the individual electronic subsystems 220 and oneor more pumps are provided within the local cooling station 600 of thecooling apparatus. In addition, the dedicated coolant-carrying tubesections of the embodiments of FIGS. 2-5B are replaced with commoncoolant flow tubing through the local cooling station. As illustrated,the local cooling station 600 includes ducting 241, 242, whichfacilitates passage of cooling airflow 244 across a liquid-to-air heatexchanger 610 of the local cooling station. In this embodiment, thelocal cooling station 600 includes a coolant distribution unit 640 whichcomprises, in one embodiment, a coolant reservoir 641 and one or morecoolant pumps 642 for pumping cooled liquid coolant via a coolant supplymanifold 620 in parallel to the individual liquid-cooled structures 530within the electronic subsystems 220 of electronics rack 210′. Heatedcoolant is exhausted via a common coolant return manifold 630 forpassage through liquid-to-air heat exchanger 610. In the embodimentillustrated, the common coolant flow tubing 611 within liquid-to-airheat exchanger 610 is oriented vertically within the heat exchanger, andthe air-cooled fins 612 are oriented substantially horizontally (by wayof example only). Quick connect couplings 513 may also be provided tofacilitate connection or disconnection of the respective liquid-cooledelectronic subsystems from the local cooling station 600.

Note that in the embodiments described herein, in operation, coolant(whether vapor or liquid) is circulated between the respective coolingsubsystems within the associated electronics rack and the liquid-to-airheat exchanger of the adjacent, local cooling station. Heat istransferred via the circulating coolant from one or more heat-generatingelectronic components within the electronic subsystem, and rejected inthe liquid-to-air heat exchanger of the respective cooling station tothe cooling airflow passing across the liquid-to-air heat exchanger. Theheated airflow is then exhausted via, for example, a common airflowexhaust plenum. Note also that, although described herein as having aone-to-one correspondence between the local cooling station and anelectronics rack, a local cooling station could be configured toaccommodate, for example, two or more electronics racks, if desired.

FIGS. 7-9 depicts various control processes of a cooling apparatus suchas a described above in connection with FIGS. 2-6C. In one embodiment,the control processes may be implemented by a controller associated withthe cooling apparatus, such as controller 236 of the cooling apparatusof FIG. 2.

Referring to the process of FIG. 7, the controller collects data on theoutdoor temperature at the air intake to the cooling apparatus 700, anddetermines whether the outdoor temperature (T_(o)) is less than a firsttemperature threshold (T_(spec1)) 705. In one embodiment, the firsttemperature threshold (T_(spec1)) is the coldest air temperatureallowable to the liquid-to-air heat exchangers of the local coolingstations without entering a winter mode. Assuming that the outdoortemperature (T_(o)) is greater than or equal to the first specifiedtemperature (T_(spec1)), then processing determines whether the outdoortemperature (T_(o)) is greater than a second specified temperature(T_(spec2)) 710, which is a threshold of the warmest air temperatureallowable to the liquid-to-air heat exchangers without entering a summermode. Assuming that the outdoor temperature (T_(o)) is less than orequal to the second specified temperature (T_(spec2)), then the datacenter is in regular operating mode 715, and the winter or summeroperating modes may be disengaged if previously engaged. Processing thenwaits a first time interval (t₁) 720 before again collecting data on theoutdoor air temperature at the intake of the cooling apparatus 700, andrepeats the process.

Assuming that the outdoor temperature (T_(o)) is less than the firstspecified temperature threshold (T_(spec1)), meaning that the outdoortemperature has dropped below the coolest allowable air temperaturethreshold to the liquid-to-air heat exchangers, then the controllerplaces the cooling apparatus in winter mode, meaning that the air inlettemperature requires heating 730. Responsive to this, processinginitiates recirculation mode to redirect a portion of the warm airflowexhausting via the airflow exhaust plenum into the cooling airflowsupply plenum 735. Processing then waits a second time interval (t₂)740, before again collecting outdoor temperature readings 700, andrepeating the process. Note that in one embodiment, time interval t₁ andtime interval t₂ may be the same time intervals, or may be differentintervals.

Assuming that the outdoor temperature (T_(o)) is greater than the secondtemperature threshold (T_(spec2)), then processing places the coolingapparatus in summer operating mode, and initiates a dry bulb temperaturedecrease of the outdoor air being drawn into the cooling airflow supplyplenum across the evaporative cooling system 750. Processing enters theevaporative cooling mode 755 to initiate evaporative cooling of theoutdoor air drawn across the evaporative cooling media of theevaporative cooling system, and then waits second time interval (t₂) 740before again collecting outdoor temperature data, and repeating theprocess.

FIGS. 8A & 8B depict one embodiment of an evaporative cooling processand evaporative cooling system, respectively, in accordance with anaspect of the present invention. Referring to the process of FIG. 8A,evaporative cooling mode 800 is entered with the controller collectingdata for controlling evaporative cooler pump ON/OFF, including inletduct air temperature (T_(in)) 810. Processing determines whether theinlet duct air temperature (T_(in)) is greater than the second specifiedtemperature (T_(spec2)) 820, and if “yes”, switches the evaporativecooling pump ON 830. As illustrated in the embodiment of FIG. 8B, theevaporative cooling system may comprise a porous media 801 through whichthe inlet air passes, a container 802, a pump 803, and one or more spraynozzles 804. Water 805 is pumped via pump 803 to water spray nozzles 804where it drips down porous media 801 and cools by evaporation the airpassing through the porous media. Once the water level falls below acertain threshold (monitored, e.g., using a float valve (not shown)),additional water can be provided to container 802 via a water supplyline 806.

Continuing with the processing 840 of FIG. 8A, if the inlet duct airtemperature (T_(in)) is less than a third specified temperaturethreshold (T_(spec3)) processing switches the pump OFF 850. Otherwise,processing waits a time interval (t) 835 before again collecting therelevant data 810, and repeating the process. Note that in this example,the third specified temperature threshold (T_(spec3)) is a defined,acceptable air temperature for the liquid-to-air heat exchangers of thelocal cooling stations.

FIG. 9 illustrates one embodiment of processing for control of theairflow recirculation mode. As noted, airflow recirculation mode isentered 900 when the outdoor temperature (T_(o)) is below a firstspecified temperature threshold. Processing initially collects data forcontrol of the recirculation fan's RPMs, including the inlet duct airtemperature and the exhaust duct air temperature 910. Processingdetermines whether the inlet duct air temperature (T_(in)) is less thanthe first temperature threshold (T_(spec1)) 920, and if “yes”, increasesthe recirculation fan's speed (RPMs) by a set ΔRPM 930. If the inletduct air temperature (T_(in)) is greater than or equal to the firsttemperature threshold (T_(spec1)), then processing determines whetherthe inlet duct air temperature (T_(in)) is greater than a fourthspecified temperature threshold (T_(spec4)). In this processing example,the inlet duct air temperature (T_(in)) is the air temperaturedownstream of the recirculation fan, and the fourth temperaturethreshold (T_(spec4)) is a defined, acceptably cool air temperature thatis allowable to the liquid-to-air heat exchanger coil of the localcooling stations. If the inlet duct air temperature (T_(in)) is greaterthan the fourth temperature threshold (T_(spec4)) 940, then therecirculation fan's speed may be reduced by the set amount (ΔRPM) 950.Thereafter, processing waits time interval (t) 935 before againcollecting temperature data for control of the recirculation fan speed,as described above.

As will be appreciated by one skilled in the art, control aspects of thepresent invention may be embodied as a system, method or computerprogram product. Accordingly, control aspects of the present inventionmay take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and hardware aspects that mayall generally be referred to herein as a “circuit,” “module” or“system”. Furthermore, control aspects of the present invention may takethe form of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible, non-transitorymedium that can contain or store a program for use by or in connectionwith an instruction execution system, apparatus, or device.

In one example, a computer program product includes, for instance, oneor more computer readable storage media to store computer readableprogram code means or logic thereon to provide and facilitate one ormore aspects of the present invention.

Program code embodied on a computer readable medium may be transmittedusing an appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programminglanguage, such as Java, Smalltalk, C++ or the like, and conventionalprocedural programming languages, such as the “C” programming language,assembler or similar programming languages.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.

What is claimed is:
 1. A method of cooling at least one electronic subsystem of an electronics rack, the method comprising: obtaining a cooling apparatus comprising: a local cooling station, the local cooling station comprising: a liquid-to-air heat exchanger; and ducting for directing cooling airflow across the liquid-to-air heat exchanger; a least one cooling subsystem for association with the at least one electronic subsystem, one cooling subsystem of the at least one cooling subsystem to provide cooling to a respective electronic subsystem of the at least one electronic subsystem, the one cooling subsystem comprising at least one of a housing facilitating immersion cooling of one or more electronic components of the respective electronic subsystem, or a liquid-cooled structure providing conductive cooling of one or more electronic components of the respective electronic subsystem; and at least one coolant loop for coupling the one cooling subsystem to the liquid-to-air heat exchanger of a respective local cooling station; disposing the electronics rack and a respective local cooling station adjacent to each other, and employing the one coolant loop to couple in fluid communication the one cooling subsystem associated with the respective electronic subsystem and the liquid-to-air heat exchanger of the respective local cooling station; and establishing cooling airflow through the ducting and across the liquid-to-air heat exchanger, and circulation of coolant through the liquid-to-air heat exchanger, the coolant loop and the one cooling subsystem, wherein heat is transferred via the circulating coolant from the respective electronic subsystem and rejected in the liquid-to-air heat exchanger of the respective local cooling station to the cooling airflow passing across the liquid-to-air heat exchanger.
 2. The method of claim 1, wherein the electronics rack comprises multiple electronic subsystems, and the cooling apparatus comprises multiple cooling subsystems, each cooling subsystem being associated with a respective electronic subsystem of the multiple electronic subsystems, and wherein the at least one coolant loop couples the multiple cooling subsystems to the liquid-to-air heat exchanger of the respective local cooling station, the at least one coolant loop facilitating circulation of coolant between the multiple cooling subsystems and the liquid-to-air heat exchanger of the respective local cooling station, and wherein heat is transferred via the circulating coolant from the multiple cooling subsystems and rejected in the liquid-to-air heat exchanger of the respective local cooling station to the cooling airflow passing across the liquid-to-air heat exchanger.
 3. The method of claim 1, wherein the cooling airflow comprises outdoor air drawn into the cooling apparatus.
 4. The method of claim 3, wherein the cooling apparatus further comprises a cooling airflow supply plenum and an airflow exhaust plenum, the ducting of the at least one local cooling station being coupled to the cooling airflow supply plenum and to the airflow exhaust plenum, and the ducting receiving cooling airflow from the cooling airflow supply plenum, directing the cooling airflow across the liquid-to-air heat exchanger, and exhausting heated airflow from the liquid-to-air heat exchanger to the airflow exhaust plenum.
 5. The method of claim 4, wherein the cooling apparatus further comprises a controllable evaporative cooling system associated with the cooling airflow supply plenum for selectively cooling outdoor air being drawn into the cooling airflow supply plenum, and a controller coupled to the controllable evaporative cooling system, the controller activating the controllable evaporative cooling system responsive to a temperature of the outdoor air exceeding a defined high temperature threshold.
 6. The method of claim 4, wherein the cooling apparatus further comprises a controllable recirculation fan for selectively recirculating a portion of exhausting airflow in the airflow exhaust plenum to the cooling airflow supply plenum, and a controller coupled to the controllable recirculation fan, the controller activating recirculation of at least a portion of the exhausting heated airflow in the airflow exhaust plenum to the cooling airflow supply plenum responsive to a temperature of the outdoor air being below a defined low temperature threshold. 